Skip to content

[improve](hive) Refactor csv reader #50379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jun 6, 2025
Merged

[improve](hive) Refactor csv reader #50379

merged 9 commits into from
Jun 6, 2025

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Apr 24, 2025 •
edited
Loading

What problem does this PR solve?

  • Impl TextReader for reading hive text table.
  • Cleanup CsvReader
  • Add cases to test csv and text format

Release note

behavior changes:

  1. Don't parse "\N" or other string as null for hive OpenCsv table
    In Hive's OpenCsv table, there is no definition for the representation of null values. This behavior is different from the behavior of importing CSV tables internally in Doris.
  2. Fix the bug when reading complex type for hive OpenCsv table
    The current code incorrectly uses the hive text format to parse the complex types of Hive OpenCsv tables. This PR fixes this behavior and uses the json format for parsing.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223 suxiaogang223 marked this pull request as draft April 24, 2025 08:04
@suxiaogang223
Copy link
Contributor Author

run buildall

1 similar comment
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34089 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5650a1d1a11b09aa9f804cf810d60aec61718c1f, data reload: false

------ Round 1 ----------------------------------
q1	25956	5101	5010	5010
q2	2066	274	188	188
q3	10409	1225	708	708
q4	10233	1004	529	529
q5	7550	3006	2391	2391
q6	185	161	130	130
q7	907	740	604	604
q8	9327	1369	1177	1177
q9	6843	5119	5181	5119
q10	6805	2285	1900	1900
q11	477	280	277	277
q12	347	360	227	227
q13	17760	3636	3065	3065
q14	220	228	210	210
q15	534	488	470	470
q16	443	450	396	396
q17	599	872	364	364
q18	8034	7113	7196	7113
q19	1215	941	538	538
q20	333	330	222	222
q21	4296	2579	2476	2476
q22	1066	1037	975	975
Total cold run time: 115605 ms
Total hot run time: 34089 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5118	5124	5117	5117
q2	239	331	227	227
q3	2175	2651	2287	2287
q4	1395	1794	1353	1353
q5	4548	4435	4412	4412
q6	213	164	124	124
q7	1960	1863	1738	1738
q8	2576	2467	2512	2467
q9	7164	7088	7148	7088
q10	3020	3166	2747	2747
q11	594	499	479	479
q12	717	781	588	588
q13	3494	3951	3345	3345
q14	294	293	265	265
q15	525	479	474	474
q16	467	483	445	445
q17	1188	1551	1397	1397
q18	7734	7428	7269	7269
q19	814	816	801	801
q20	1965	2007	1937	1937
q21	5202	4733	4594	4594
q22	1054	1049	1010	1010
Total cold run time: 52456 ms
Total hot run time: 50164 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184641 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5650a1d1a11b09aa9f804cf810d60aec61718c1f, data reload: false

query1	1032	454	499	454
query2	6662	1852	1870	1852
query3	6764	222	213	213
query4	26305	23659	22917	22917
query5	4333	607	471	471
query6	321	209	217	209
query7	4625	486	289	289
query8	297	243	236	236
query9	8652	2523	2546	2523
query10	485	289	251	251
query11	16010	14960	14776	14776
query12	165	108	109	108
query13	1675	522	389	389
query14	9412	6179	6159	6159
query15	201	185	172	172
query16	7252	635	444	444
query17	1203	719	576	576
query18	1985	405	302	302
query19	190	183	160	160
query20	119	120	120	120
query21	214	133	109	109
query22	4068	4165	3962	3962
query23	33966	32945	33119	32945
query24	8405	2361	2370	2361
query25	539	486	374	374
query26	1236	263	149	149
query27	2756	508	325	325
query28	4340	2092	2068	2068
query29	764	581	425	425
query30	281	211	182	182
query31	931	864	754	754
query32	72	69	69	69
query33	559	377	303	303
query34	795	847	507	507
query35	769	787	742	742
query36	929	992	910	910
query37	113	98	77	77
query38	4198	4121	4077	4077
query39	1473	1387	1390	1387
query40	211	123	107	107
query41	55	53	55	53
query42	117	103	106	103
query43	516	490	462	462
query44	1311	789	798	789
query45	177	175	164	164
query46	817	1015	620	620
query47	1724	1792	1708	1708
query48	370	398	308	308
query49	771	483	407	407
query50	636	671	399	399
query51	4125	4055	4074	4055
query52	107	111	96	96
query53	217	252	179	179
query54	578	597	494	494
query55	82	78	82	78
query56	291	294	293	293
query57	1141	1146	1050	1050
query58	261	259	246	246
query59	2614	2591	2546	2546
query60	322	305	291	291
query61	129	131	127	127
query62	809	705	679	679
query63	229	185	182	182
query64	4297	1013	719	719
query65	4355	4245	4231	4231
query66	1133	449	306	306
query67	15695	15406	15155	15155
query68	8675	880	522	522
query69	467	300	261	261
query70	1195	1140	1096	1096
query71	487	324	291	291
query72	5511	4672	4688	4672
query73	732	607	338	338
query74	8866	8850	8967	8850
query75	4031	3202	2703	2703
query76	3709	1182	743	743
query77	789	377	287	287
query78	9933	10231	9231	9231
query79	1907	798	563	563
query80	591	523	449	449
query81	476	250	217	217
query82	435	123	94	94
query83	260	248	233	233
query84	243	111	85	85
query85	782	357	315	315
query86	346	268	283	268
query87	4344	4476	4344	4344
query88	3182	2170	2154	2154
query89	383	317	288	288
query90	1937	207	206	206
query91	139	141	111	111
query92	76	62	61	61
query93	1065	933	581	581
query94	670	399	313	313
query95	373	294	288	288
query96	476	563	268	268
query97	3149	3210	3125	3125
query98	223	201	198	198
query99	1436	1414	1292	1292
Total cold run time: 274074 ms
Total hot run time: 184641 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5650a1d1a11b09aa9f804cf810d60aec61718c1f, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.11	0.11
query3	0.26	0.19	0.19
query4	1.60	0.18	0.20
query5	0.58	0.59	0.61
query6	1.19	0.71	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.56	0.56	0.56
query11	0.16	0.10	0.12
query12	0.15	0.11	0.12
query13	0.61	0.59	0.59
query14	1.19	1.16	1.19
query15	0.88	0.84	0.84
query16	0.40	0.38	0.38
query17	1.05	1.03	1.00
query18	0.21	0.21	0.20
query19	1.92	1.81	1.79
query20	0.01	0.02	0.01
query21	15.40	0.89	0.56
query22	0.76	1.19	0.77
query23	14.82	1.38	0.62
query24	7.24	1.71	0.32
query25	0.28	0.14	0.13
query26	0.64	0.16	0.15
query27	0.05	0.04	0.04
query28	9.12	0.88	0.42
query29	12.55	4.01	3.36
query30	0.25	0.09	0.07
query31	2.82	0.58	0.38
query32	3.24	0.54	0.48
query33	3.03	3.04	3.01
query34	15.76	5.11	4.50
query35	4.51	4.60	4.48
query36	0.65	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.17	0.16	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.22 s
Total hot run time: 29.18 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 4.55% (13/286) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.82% (14621/27166)
Line Coverage 42.61% (126881/297768)
Region Coverage 41.41% (64870/156641)
Branch Coverage 35.96% (32613/90692)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 4.55% (13/286) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage
Line Coverage
Region Coverage
Branch Coverage

@suxiaogang223
Copy link
Contributor Author

run external

1 similar comment
@suxiaogang223
Copy link
Contributor Author

run external

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.06% (1098/1322)
Line Coverage: 65.83% (18372/27909)
Region Coverage: 65.31% (9113/13954)
Branch Coverage: 55.19% (4895/8870)
Coverage Report: http://coverage.selectdb-in.cc/coverage/52a7949fc04539b6896be4dc0a35c46db90b026a_52a7949fc04539b6896be4dc0a35c46db90b026a_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 34204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52a7949fc04539b6896be4dc0a35c46db90b026a, data reload: false

------ Round 1 ----------------------------------
q1	26328	5065	5027	5027
q2	2088	277	198	198
q3	10572	1258	719	719
q4	10245	1001	539	539
q5	7880	2400	2367	2367
q6	192	164	132	132
q7	933	749	606	606
q8	9336	1287	1140	1140
q9	6995	5127	5220	5127
q10	6840	2327	1896	1896
q11	475	293	275	275
q12	352	358	221	221
q13	17787	3685	3113	3113
q14	226	225	208	208
q15	536	494	507	494
q16	459	449	404	404
q17	607	887	359	359
q18	7904	7164	7180	7164
q19	1591	971	585	585
q20	334	342	223	223
q21	3941	3357	2446	2446
q22	1044	1025	961	961
Total cold run time: 116665 ms
Total hot run time: 34204 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5159	5090	5096	5090
q2	236	324	222	222
q3	2167	2656	2318	2318
q4	1360	1816	1451	1451
q5	4586	4404	4283	4283
q6	212	167	123	123
q7	1994	1914	1746	1746
q8	2630	2605	2482	2482
q9	7273	7277	7178	7178
q10	3004	3227	2733	2733
q11	567	494	495	494
q12	675	787	608	608
q13	3427	3902	3260	3260
q14	288	292	261	261
q15	532	484	472	472
q16	458	504	468	468
q17	1172	1552	1447	1447
q18	7626	7520	7554	7520
q19	827	832	981	832
q20	1971	2029	1879	1879
q21	5032	4692	4489	4489
q22	1082	1023	987	987
Total cold run time: 52278 ms
Total hot run time: 50343 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185210 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52a7949fc04539b6896be4dc0a35c46db90b026a, data reload: false

query1	1016	473	520	473
query2	6569	1803	1749	1749
query3	6751	230	216	216
query4	26090	23295	23422	23295
query5	4329	606	452	452
query6	292	215	200	200
query7	4622	495	280	280
query8	300	257	244	244
query9	8658	2575	2582	2575
query10	472	308	263	263
query11	15567	15175	14781	14781
query12	161	121	112	112
query13	1671	525	405	405
query14	8749	5960	6468	5960
query15	204	186	170	170
query16	7181	632	473	473
query17	1190	688	560	560
query18	1974	387	293	293
query19	188	200	152	152
query20	119	118	115	115
query21	206	128	112	112
query22	3976	4181	4065	4065
query23	33824	32748	32949	32748
query24	8466	2356	2350	2350
query25	534	466	394	394
query26	1236	278	168	168
query27	2732	504	327	327
query28	4346	2115	2097	2097
query29	762	551	421	421
query30	281	213	196	196
query31	931	854	727	727
query32	73	68	65	65
query33	551	357	318	318
query34	782	863	513	513
query35	787	816	732	732
query36	958	994	886	886
query37	111	102	74	74
query38	4172	4208	4065	4065
query39	1439	1382	1409	1382
query40	217	121	105	105
query41	61	54	52	52
query42	122	104	114	104
query43	508	501	458	458
query44	1281	812	804	804
query45	173	174	169	169
query46	817	1023	616	616
query47	1773	1809	1740	1740
query48	367	410	305	305
query49	781	484	414	414
query50	634	666	408	408
query51	4161	4142	4070	4070
query52	116	111	97	97
query53	228	254	180	180
query54	581	569	500	500
query55	80	78	78	78
query56	296	289	291	289
query57	1137	1136	1088	1088
query58	267	265	261	261
query59	2579	2688	2570	2570
query60	335	347	330	330
query61	152	150	152	150
query62	809	728	655	655
query63	227	195	186	186
query64	4502	1074	673	673
query65	4310	4178	4161	4161
query66	1198	432	305	305
query67	15901	15505	15258	15258
query68	7902	880	534	534
query69	475	297	263	263
query70	1171	1116	1059	1059
query71	479	315	282	282
query72	5703	4770	4820	4770
query73	687	631	357	357
query74	8790	9236	8902	8902
query75	3714	3183	2788	2788
query76	3616	1190	758	758
query77	804	378	289	289
query78	9922	9954	9305	9305
query79	2416	801	568	568
query80	695	557	438	438
query81	501	251	210	210
query82	471	130	103	103
query83	287	259	225	225
query84	287	104	89	89
query85	787	362	316	316
query86	387	294	301	294
query87	4308	4359	4281	4281
query88	3747	2186	2198	2186
query89	385	319	283	283
query90	1832	209	222	209
query91	147	145	111	111
query92	77	62	60	60
query93	2129	955	584	584
query94	683	415	315	315
query95	371	295	287	287
query96	484	562	271	271
query97	3142	3270	3157	3157
query98	248	202	210	202
query99	1457	1378	1280	1280
Total cold run time: 273987 ms
Total hot run time: 185210 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52a7949fc04539b6896be4dc0a35c46db90b026a, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.26	0.20	0.19
query4	1.59	0.20	0.11
query5	0.56	0.56	0.54
query6	1.21	0.71	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.52	0.52
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.59	0.59
query14	1.21	1.22	1.21
query15	0.88	0.85	0.84
query16	0.37	0.38	0.39
query17	1.00	1.02	1.04
query18	0.22	0.20	0.19
query19	1.92	1.79	1.84
query20	0.01	0.01	0.01
query21	15.39	0.94	0.55
query22	0.75	1.18	0.62
query23	14.98	1.38	0.60
query24	7.10	1.37	0.49
query25	0.49	0.35	0.06
query26	0.53	0.17	0.15
query27	0.05	0.06	0.05
query28	9.59	0.87	0.43
query29	12.54	4.03	3.34
query30	0.25	0.10	0.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.47
query33	3.12	3.01	3.02
query34	15.72	5.13	4.51
query35	4.54	4.50	4.52
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.77 s
Total hot run time: 29.1 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 4.58% (13/284) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.44% (14766/27122)
Line Coverage 43.52% (129207/296900)
Region Coverage 42.22% (65988/156296)
Branch Coverage 36.75% (33245/90470)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 4.55% (13/286) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.52% (14782/26626)
Line Coverage 45.14% (133795/296416)
Region Coverage 42.07% (76893/182753)
Branch Coverage 36.08% (37174/103044)

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.06% (1098/1322)
Line Coverage: 65.89% (18388/27909)
Region Coverage: 65.29% (9110/13954)
Branch Coverage: 55.24% (4900/8870)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a772c870025b13d35a08d10543510c94f52d3bf_7a772c870025b13d35a08d10543510c94f52d3bf_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 33981 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a772c870025b13d35a08d10543510c94f52d3bf, data reload: false

------ Round 1 ----------------------------------
q1	26310	5012	4971	4971
q2	2093	280	179	179
q3	10579	1276	697	697
q4	10235	1022	536	536
q5	7845	2390	2386	2386
q6	185	161	132	132
q7	915	749	623	623
q8	9318	1333	1119	1119
q9	6788	5042	5112	5042
q10	6826	2297	1875	1875
q11	490	280	273	273
q12	346	359	215	215
q13	17768	3676	3108	3108
q14	232	229	221	221
q15	530	474	513	474
q16	429	440	384	384
q17	599	876	365	365
q18	7662	7130	7223	7130
q19	2391	980	559	559
q20	318	320	216	216
q21	3947	3431	2512	2512
q22	1051	1029	964	964
Total cold run time: 116857 ms
Total hot run time: 33981 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5212	5039	5053	5039
q2	241	329	230	230
q3	2178	2695	2356	2356
q4	1407	1805	1497	1497
q5	4478	4419	4412	4412
q6	222	167	130	130
q7	2011	1921	1768	1768
q8	2627	2574	2569	2569
q9	7200	7080	7200	7080
q10	2986	3168	2778	2778
q11	563	524	504	504
q12	668	769	643	643
q13	3534	3949	3352	3352
q14	283	309	263	263
q15	532	489	488	488
q16	448	477	440	440
q17	1164	1595	1356	1356
q18	7712	7535	7529	7529
q19	829	868	1051	868
q20	1970	1963	1864	1864
q21	5311	4728	4743	4728
q22	1116	1062	1035	1035
Total cold run time: 52692 ms
Total hot run time: 50929 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7a772c870025b13d35a08d10543510c94f52d3bf, data reload: false

query1	1459	1089	1087	1087
query2	6210	1820	1741	1741
query3	11022	4499	4537	4499
query4	55751	25806	22985	22985
query5	5179	501	475	475
query6	369	191	188	188
query7	5112	496	285	285
query8	335	260	245	245
query9	6628	2549	2575	2549
query10	416	318	269	269
query11	15187	14980	15126	14980
query12	160	112	101	101
query13	1172	503	396	396
query14	10148	6391	6511	6391
query15	215	206	177	177
query16	7018	655	511	511
query17	1105	721	561	561
query18	1551	387	311	311
query19	206	191	161	161
query20	127	117	126	117
query21	203	127	109	109
query22	4433	4398	4405	4398
query23	34228	33537	33624	33537
query24	6586	2476	2437	2437
query25	505	501	445	445
query26	679	293	158	158
query27	2226	506	341	341
query28	3213	2119	2124	2119
query29	557	558	419	419
query30	268	213	191	191
query31	851	857	761	761
query32	79	140	64	64
query33	436	376	312	312
query34	765	848	530	530
query35	810	837	746	746
query36	962	1008	931	931
query37	118	103	77	77
query38	4265	4217	4163	4163
query39	1493	1440	1424	1424
query40	215	122	110	110
query41	55	53	54	53
query42	125	112	113	112
query43	499	506	469	469
query44	1317	807	805	805
query45	190	179	171	171
query46	849	1016	682	682
query47	1850	1896	1833	1833
query48	388	410	313	313
query49	693	494	429	429
query50	652	688	402	402
query51	4193	4338	4225	4225
query52	120	101	99	99
query53	228	263	185	185
query54	590	604	523	523
query55	89	82	86	82
query56	304	326	281	281
query57	1165	1208	1122	1122
query58	271	274	256	256
query59	2629	2732	2588	2588
query60	333	324	326	324
query61	130	135	126	126
query62	737	741	692	692
query63	232	183	191	183
query64	1466	1062	765	765
query65	4447	4340	4235	4235
query66	715	452	305	305
query67	16011	15571	15561	15561
query68	7264	875	522	522
query69	554	296	260	260
query70	1157	1070	1055	1055
query71	526	310	281	281
query72	5764	4779	4889	4779
query73	1166	680	343	343
query74	8973	9025	8645	8645
query75	3797	3238	2713	2713
query76	4228	1199	744	744
query77	625	376	276	276
query78	10195	10029	9283	9283
query79	2579	813	567	567
query80	678	511	446	446
query81	489	258	213	213
query82	582	128	100	100
query83	251	250	250	250
query84	302	108	86	86
query85	890	356	317	317
query86	401	307	258	258
query87	4409	4340	4371	4340
query88	3367	2204	2182	2182
query89	409	328	284	284
query90	1791	212	215	212
query91	140	140	109	109
query92	75	63	60	60
query93	2545	951	577	577
query94	680	407	290	290
query95	368	299	291	291
query96	488	567	272	272
query97	3177	3219	3229	3219
query98	221	214	198	198
query99	1422	1404	1311	1311
Total cold run time: 302053 ms
Total hot run time: 192605 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a772c870025b13d35a08d10543510c94f52d3bf, data reload: false

query1	0.04	0.03	0.03
query2	0.15	0.10	0.10
query3	0.34	0.20	0.20
query4	1.60	0.21	0.19
query5	0.63	0.60	0.61
query6	1.19	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.62	0.52	0.53
query10	0.59	0.59	0.58
query11	0.26	0.12	0.12
query12	0.25	0.13	0.14
query13	0.64	0.63	0.62
query14	0.80	0.83	0.82
query15	0.95	0.88	0.87
query16	0.38	0.37	0.38
query17	1.05	1.00	1.04
query18	0.20	0.18	0.18
query19	1.92	1.81	1.83
query20	0.02	0.01	0.02
query21	15.40	0.96	0.68
query22	0.93	1.01	0.82
query23	14.68	1.58	0.76
query24	5.53	0.54	0.29
query25	0.16	0.09	0.08
query26	0.56	0.21	0.18
query27	0.08	0.08	0.09
query28	11.04	1.15	0.57
query29	12.53	4.05	3.41
query30	0.28	0.08	0.07
query31	2.82	0.63	0.42
query32	3.23	0.58	0.50
query33	3.08	3.07	3.09
query34	16.45	5.10	4.44
query35	4.46	4.50	4.47
query36	0.65	0.50	0.49
query37	0.19	0.16	0.16
query38	0.17	0.17	0.17
query39	0.05	0.04	0.05
query40	0.19	0.17	0.15
query41	0.11	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 104.39 s
Total hot run time: 29.92 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 4.58% (13/284) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.73% (14763/26976)
Line Coverage 43.79% (129282/295248)
Region Coverage 42.52% (66008/155249)
Branch Coverage 37.09% (33263/89686)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 18.31% (52/284) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.02% (20660/26480)
Line Coverage 71.57% (210965/294764)
Region Coverage 69.76% (126774/181718)
Branch Coverage 63.01% (64440/102268)

@suxiaogang223
Copy link
Contributor Author

run buildall

@suxiaogang223 suxiaogang223 marked this pull request as ready for review April 29, 2025 02:10
@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1100/1324)
Line Coverage: 65.88% (18400/27928)
Region Coverage: 65.37% (9128/13964)
Branch Coverage: 55.25% (4903/8874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1fe74eb55e604067de350b2c01a57f79127b711e_1fe74eb55e604067de350b2c01a57f79127b711e_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 34120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fc3e4adc9036c348a83a0cad346689ccb7096b46, data reload: false

------ Round 1 ----------------------------------
q1	26914	5014	5009	5009
q2	1981	304	189	189
q3	10350	1276	726	726
q4	10242	1026	540	540
q5	7632	2361	2358	2358
q6	187	167	135	135
q7	904	766	624	624
q8	9312	1296	1130	1130
q9	6826	5211	5104	5104
q10	6888	2336	1910	1910
q11	478	306	311	306
q12	349	355	231	231
q13	17786	3696	3088	3088
q14	239	239	219	219
q15	570	474	487	474
q16	442	436	380	380
q17	629	861	382	382
q18	8057	7177	7201	7177
q19	1821	980	563	563
q20	343	342	224	224
q21	3843	3171	2372	2372
q22	1030	1009	979	979
Total cold run time: 116823 ms
Total hot run time: 34120 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5163	5096	5083	5083
q2	236	330	220	220
q3	2160	2661	2352	2352
q4	1309	1804	1440	1440
q5	4564	4449	4401	4401
q6	211	161	135	135
q7	2052	1904	1740	1740
q8	2743	2672	2568	2568
q9	7257	7259	7162	7162
q10	3015	3219	2782	2782
q11	586	508	489	489
q12	683	793	606	606
q13	3519	3922	3449	3449
q14	300	301	288	288
q15	544	507	491	491
q16	467	501	438	438
q17	1181	1572	1390	1390
q18	7795	7412	7420	7412
q19	855	840	924	840
q20	1996	2014	1901	1901
q21	4839	4490	4374	4374
q22	1089	1073	1038	1038
Total cold run time: 52564 ms
Total hot run time: 50599 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fc3e4adc9036c348a83a0cad346689ccb7096b46, data reload: false

query1	1440	1100	1068	1068
query2	6269	1862	1883	1862
query3	11065	4596	4397	4397
query4	52903	25922	22963	22963
query5	5148	642	479	479
query6	351	223	209	209
query7	4928	517	295	295
query8	299	232	221	221
query9	5736	2773	2778	2773
query10	464	337	271	271
query11	15056	15040	14900	14900
query12	160	110	117	110
query13	1078	549	429	429
query14	10188	6532	6436	6436
query15	209	185	184	184
query16	7087	631	501	501
query17	1095	719	584	584
query18	1597	410	326	326
query19	212	191	176	176
query20	133	138	123	123
query21	207	124	113	113
query22	4479	4462	4317	4317
query23	34205	33512	33809	33512
query24	6650	2405	2442	2405
query25	462	484	408	408
query26	701	277	163	163
query27	2379	556	361	361
query28	3060	2291	2259	2259
query29	598	552	459	459
query30	271	229	197	197
query31	884	867	775	775
query32	70	67	65	65
query33	501	378	337	337
query34	808	867	550	550
query35	790	875	755	755
query36	973	1012	930	930
query37	118	107	79	79
query38	4382	4348	4167	4167
query39	1533	1531	1466	1466
query40	216	126	109	109
query41	61	59	67	59
query42	126	112	109	109
query43	515	514	505	505
query44	1346	860	884	860
query45	186	182	178	178
query46	867	1050	662	662
query47	1854	1904	1787	1787
query48	460	453	349	349
query49	650	500	419	419
query50	703	709	420	420
query51	4373	4280	4221	4221
query52	111	113	105	105
query53	235	266	187	187
query54	601	604	522	522
query55	89	88	91	88
query56	316	315	322	315
query57	1178	1186	1150	1150
query58	272	264	271	264
query59	2702	2812	2818	2812
query60	342	335	315	315
query61	134	130	136	130
query62	751	753	687	687
query63	237	198	197	197
query64	2045	1043	704	704
query65	4206	4175	4141	4141
query66	723	400	309	309
query67	16022	15555	15444	15444
query68	6055	908	555	555
query69	543	317	289	289
query70	1267	1151	1129	1129
query71	451	323	301	301
query72	5960	4731	4775	4731
query73	1412	629	368	368
query74	9247	8824	9101	8824
query75	3189	3252	2704	2704
query76	3908	1203	773	773
query77	536	372	296	296
query78	10035	10227	9309	9309
query79	2712	842	592	592
query80	663	532	448	448
query81	499	256	217	217
query82	446	134	101	101
query83	311	267	236	236
query84	302	113	94	94
query85	785	365	334	334
query86	354	328	295	295
query87	4403	4593	4446	4446
query88	3597	2320	2307	2307
query89	395	329	286	286
query90	1656	207	203	203
query91	152	151	114	114
query92	76	66	58	58
query93	2060	966	608	608
query94	730	408	291	291
query95	375	306	297	297
query96	525	583	291	291
query97	2722	2784	2616	2616
query98	235	216	215	215
query99	1445	1389	1277	1277
Total cold run time: 296856 ms
Total hot run time: 193278 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fc3e4adc9036c348a83a0cad346689ccb7096b46, data reload: false

query1	0.04	0.03	0.03
query2	0.16	0.11	0.10
query3	0.35	0.20	0.19
query4	1.61	0.22	0.09
query5	0.43	0.41	0.41
query6	1.16	0.66	0.68
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.63	0.52	0.53
query10	0.59	0.58	0.58
query11	0.28	0.13	0.14
query12	0.26	0.13	0.14
query13	0.64	0.62	0.62
query14	0.83	0.82	0.82
query15	0.98	0.90	0.88
query16	0.37	0.38	0.38
query17	1.10	1.05	1.05
query18	0.20	0.18	0.17
query19	2.00	1.83	1.93
query20	0.02	0.02	0.02
query21	15.40	0.97	0.67
query22	0.93	1.06	0.86
query23	14.66	1.53	0.77
query24	5.43	0.58	0.29
query25	0.16	0.09	0.09
query26	0.56	0.24	0.18
query27	0.08	0.08	0.08
query28	11.04	1.22	0.61
query29	12.53	4.09	3.44
query30	0.29	0.08	0.06
query31	2.84	0.62	0.43
query32	3.25	0.61	0.50
query33	3.16	3.15	3.09
query34	16.60	5.12	4.41
query35	4.50	4.48	4.44
query36	0.63	0.52	0.50
query37	0.20	0.17	0.18
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.21	0.16	0.15
query41	0.11	0.05	0.05
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 104.65 s
Total hot run time: 29.76 s

@suxiaogang223
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.32% (1119/1343)
Line Coverage 66.81% (19195/28732)
Region Coverage 66.47% (9507/14302)
Branch Coverage 56.41% (5154/9136)

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 47.06% (24/51) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 33678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c9c389c6bb477c3d66dea7cc5f7a19bb7fc9c19, data reload: false

------ Round 1 ----------------------------------
q1	26115	5084	5031	5031
q2	1958	271	175	175
q3	10420	1231	708	708
q4	10236	977	515	515
q5	7672	2401	2308	2308
q6	186	167	131	131
q7	917	729	601	601
q8	9319	1287	1077	1077
q9	6709	5083	5068	5068
q10	6865	2313	1888	1888
q11	472	290	279	279
q12	351	348	209	209
q13	17768	3713	3096	3096
q14	237	239	223	223
q15	559	484	483	483
q16	432	436	377	377
q17	612	852	371	371
q18	7697	7213	7035	7035
q19	1853	981	572	572
q20	340	329	223	223
q21	3662	2596	2328	2328
q22	1064	1027	980	980
Total cold run time: 115444 ms
Total hot run time: 33678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5199	5115	5108	5108
q2	244	316	222	222
q3	2188	2670	2326	2326
q4	1338	1754	1347	1347
q5	4424	4381	4386	4381
q6	226	168	128	128
q7	1984	1916	1798	1798
q8	2579	2644	2511	2511
q9	6955	6808	6746	6746
q10	2928	3105	2662	2662
q11	569	497	486	486
q12	641	746	571	571
q13	3352	3692	3134	3134
q14	265	287	269	269
q15	519	463	474	463
q16	434	467	433	433
q17	1127	1534	1319	1319
q18	7332	7403	7593	7403
q19	791	833	929	833
q20	1958	2092	1968	1968
q21	4917	4549	4434	4434
q22	1147	1051	1019	1019
Total cold run time: 51117 ms
Total hot run time: 49561 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c9c389c6bb477c3d66dea7cc5f7a19bb7fc9c19, data reload: false

query1	1419	1095	1054	1054
query2	6275	1840	1822	1822
query3	8749	210	208	208
query4	54140	24244	22992	22992
query5	5008	502	456	456
query6	336	208	190	190
query7	5019	511	280	280
query8	276	221	199	199
query9	5428	2654	2674	2654
query10	468	347	260	260
query11	14907	15097	14898	14898
query12	172	111	111	111
query13	1057	519	431	431
query14	10234	6160	6418	6160
query15	235	218	183	183
query16	7220	658	508	508
query17	1087	727	581	581
query18	1723	432	325	325
query19	201	185	180	180
query20	133	128	123	123
query21	225	152	107	107
query22	5138	5333	4442	4442
query23	34268	33822	33671	33671
query24	6492	2446	2423	2423
query25	480	463	426	426
query26	711	279	152	152
query27	2246	508	357	357
query28	3130	2190	2181	2181
query29	575	563	437	437
query30	216	223	195	195
query31	874	851	762	762
query32	74	68	62	62
query33	477	371	338	338
query34	770	884	511	511
query35	788	828	760	760
query36	952	1018	919	919
query37	116	100	75	75
query38	4355	4358	4336	4336
query39	1528	1486	1462	1462
query40	169	118	107	107
query41	59	55	58	55
query42	131	105	106	105
query43	485	525	478	478
query44	1323	851	841	841
query45	186	181	174	174
query46	840	1041	637	637
query47	1860	1881	1858	1858
query48	405	429	320	320
query49	653	495	413	413
query50	680	699	411	411
query51	4281	4254	4277	4254
query52	104	106	109	106
query53	230	257	182	182
query54	590	589	511	511
query55	86	83	81	81
query56	332	300	296	296
query57	1150	1191	1129	1129
query58	276	279	253	253
query59	2660	2723	2658	2658
query60	323	332	313	313
query61	132	120	124	120
query62	783	753	677	677
query63	229	192	185	185
query64	1678	1088	694	694
query65	4283	4268	4316	4268
query66	739	402	312	312
query67	16345	15753	15514	15514
query68	7224	833	517	517
query69	556	294	271	271
query70	1257	1158	1103	1103
query71	493	324	307	307
query72	5365	4793	4948	4793
query73	1510	658	350	350
query74	8923	9244	8688	8688
query75	3792	3209	2673	2673
query76	4180	1203	761	761
query77	566	363	276	276
query78	9956	10168	9439	9439
query79	1545	803	580	580
query80	602	519	444	444
query81	464	261	223	223
query82	173	125	97	97
query83	347	254	233	233
query84	299	111	97	97
query85	744	354	319	319
query86	337	308	289	289
query87	4430	4566	4328	4328
query88	2919	2265	2246	2246
query89	375	321	283	283
query90	2046	204	201	201
query91	146	142	111	111
query92	76	59	57	57
query93	1114	925	580	580
query94	702	393	307	307
query95	367	291	284	284
query96	509	553	339	339
query97	2724	2796	2629	2629
query98	216	207	208	207
query99	1415	1390	1319	1319
Total cold run time: 293886 ms
Total hot run time: 188534 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c9c389c6bb477c3d66dea7cc5f7a19bb7fc9c19, data reload: false

query1	0.04	0.04	0.03
query2	0.15	0.11	0.10
query3	0.35	0.20	0.20
query4	1.61	0.20	0.08
query5	0.43	0.41	0.43
query6	1.15	0.65	0.67
query7	0.03	0.02	0.02
query8	0.06	0.05	0.04
query9	0.63	0.52	0.51
query10	0.59	0.60	0.59
query11	0.26	0.12	0.13
query12	0.26	0.13	0.14
query13	0.63	0.61	0.62
query14	0.81	0.82	0.82
query15	0.97	0.88	0.89
query16	0.36	0.38	0.37
query17	1.07	1.03	1.06
query18	0.19	0.18	0.17
query19	2.01	1.82	1.92
query20	0.02	0.01	0.02
query21	15.41	0.98	0.67
query22	0.92	1.06	0.82
query23	14.69	1.59	0.76
query24	5.24	0.61	0.30
query25	0.17	0.09	0.08
query26	0.56	0.22	0.18
query27	0.08	0.08	0.09
query28	11.07	1.17	0.59
query29	12.60	4.17	3.46
query30	0.29	0.09	0.06
query31	2.81	0.63	0.43
query32	3.24	0.60	0.52
query33	3.03	3.08	3.16
query34	16.95	5.10	4.45
query35	4.59	4.52	4.51
query36	0.64	0.50	0.49
query37	0.20	0.18	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.19	0.16	0.16
query41	0.12	0.05	0.05
query42	0.06	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 104.76 s
Total hot run time: 29.74 s

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.32% (1119/1343)
Line Coverage 66.80% (19194/28732)
Region Coverage 66.44% (9502/14302)
Branch Coverage 56.43% (5155/9136)

@doris-robot
Copy link

TPC-H: Total hot run time: 33865 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d3d0109fc32ee4e564be68bd5c289a755438b30, data reload: false

------ Round 1 ----------------------------------
q1	26134	5106	5090	5090
q2	1960	285	177	177
q3	10281	1262	712	712
q4	10213	1010	524	524
q5	7557	2357	2361	2357
q6	200	164	133	133
q7	900	756	611	611
q8	9303	1260	1039	1039
q9	6932	5096	5166	5096
q10	6834	2338	1912	1912
q11	489	294	278	278
q12	358	357	223	223
q13	17758	3757	3063	3063
q14	221	227	212	212
q15	546	490	511	490
q16	442	429	371	371
q17	603	874	372	372
q18	7521	7249	7009	7009
q19	1089	962	585	585
q20	353	353	230	230
q21	3810	3185	2394	2394
q22	1054	1031	987	987
Total cold run time: 114558 ms
Total hot run time: 33865 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5175	5085	5189	5085
q2	240	316	220	220
q3	2183	2638	2324	2324
q4	1350	1784	1341	1341
q5	4540	4379	4415	4379
q6	219	173	128	128
q7	1979	1869	1755	1755
q8	2593	2793	2642	2642
q9	7186	7103	7135	7103
q10	2981	3189	2814	2814
q11	582	505	488	488
q12	707	765	624	624
q13	3498	3928	3287	3287
q14	275	294	273	273
q15	534	474	468	468
q16	430	486	459	459
q17	1168	1461	1374	1374
q18	7784	7516	7459	7459
q19	869	886	901	886
q20	2000	2009	1873	1873
q21	4980	4460	4349	4349
q22	1075	1025	997	997
Total cold run time: 52348 ms
Total hot run time: 50328 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186043 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d3d0109fc32ee4e564be68bd5c289a755438b30, data reload: false

query1	1008	474	483	474
query2	6551	1879	1803	1803
query3	6753	226	221	221
query4	26356	23772	22844	22844
query5	4326	670	470	470
query6	306	213	219	213
query7	4623	512	296	296
query8	264	240	221	221
query9	8604	2644	2648	2644
query10	485	352	278	278
query11	15278	15052	14866	14866
query12	178	112	110	110
query13	1633	553	405	405
query14	8719	6001	6088	6001
query15	196	193	167	167
query16	7155	673	492	492
query17	933	756	558	558
query18	1955	405	289	289
query19	192	184	162	162
query20	123	120	121	120
query21	211	132	108	108
query22	4218	4267	4217	4217
query23	34156	33122	33110	33110
query24	8630	2396	2404	2396
query25	552	459	420	420
query26	1233	285	157	157
query27	2748	521	340	340
query28	4303	2148	2115	2115
query29	760	595	430	430
query30	284	223	190	190
query31	943	864	758	758
query32	76	67	63	63
query33	545	367	308	308
query34	791	921	525	525
query35	792	837	741	741
query36	1009	1023	884	884
query37	122	107	82	82
query38	4111	4240	4019	4019
query39	1472	1438	1453	1438
query40	217	126	116	116
query41	70	65	67	65
query42	140	121	110	110
query43	507	527	469	469
query44	1370	827	828	827
query45	186	183	179	179
query46	898	1103	626	626
query47	1814	1811	1733	1733
query48	395	467	317	317
query49	777	533	422	422
query50	677	744	418	418
query51	4140	4120	4111	4111
query52	114	112	103	103
query53	229	267	200	200
query54	600	593	518	518
query55	98	86	86	86
query56	329	308	319	308
query57	1191	1157	1085	1085
query58	266	263	258	258
query59	2589	2676	2492	2492
query60	320	330	314	314
query61	125	123	122	122
query62	851	768	675	675
query63	227	186	184	184
query64	4402	998	681	681
query65	4339	4138	4176	4138
query66	1138	446	312	312
query67	15842	15829	15513	15513
query68	8559	908	549	549
query69	486	314	270	270
query70	1241	1152	1096	1096
query71	455	333	302	302
query72	5359	4731	4666	4666
query73	701	586	355	355
query74	8891	9153	9246	9153
query75	3951	3318	2710	2710
query76	3627	1316	800	800
query77	799	388	301	301
query78	10253	10094	9336	9336
query79	2798	862	604	604
query80	748	539	448	448
query81	484	271	216	216
query82	540	139	95	95
query83	291	259	233	233
query84	297	109	91	91
query85	807	360	340	340
query86	391	302	285	285
query87	4427	4502	4339	4339
query88	3719	2277	2278	2277
query89	431	321	283	283
query90	1837	209	207	207
query91	151	140	113	113
query92	78	71	60	60
query93	1966	959	591	591
query94	655	446	309	309
query95	373	294	287	287
query96	514	618	288	288
query97	2728	2809	2631	2631
query98	251	208	199	199
query99	1462	1498	1292	1292
Total cold run time: 275715 ms
Total hot run time: 186043 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d3d0109fc32ee4e564be68bd5c289a755438b30, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.10
query3	0.26	0.19	0.20
query4	1.60	0.19	0.11
query5	0.42	0.41	0.42
query6	1.19	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.52	0.51
query10	0.56	0.56	0.57
query11	0.16	0.10	0.12
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	0.79	0.80	0.82
query15	0.89	0.87	0.86
query16	0.38	0.39	0.37
query17	1.02	1.06	1.04
query18	0.23	0.22	0.21
query19	1.92	1.78	1.90
query20	0.01	0.02	0.01
query21	15.40	0.90	0.55
query22	0.77	1.16	0.62
query23	15.06	1.39	0.65
query24	6.54	1.25	0.29
query25	0.34	0.29	0.06
query26	0.48	0.17	0.13
query27	0.06	0.06	0.05
query28	9.86	0.98	0.46
query29	12.55	3.96	3.37
query30	0.25	0.10	0.07
query31	2.81	0.60	0.39
query32	3.24	0.57	0.46
query33	3.05	3.05	3.09
query34	15.92	5.15	4.56
query35	4.56	4.58	4.57
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.05	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.02 s
Total hot run time: 28.58 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 47.06% (24/51) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 14.44% (39/270) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.30% (15094/26810)
Line Coverage 45.10% (134678/298640)
Region Coverage 44.20% (67733/153239)
Branch Coverage 38.76% (34715/89562)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 88.10% (237/269) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.57% (20999/26390)
Line Coverage 72.64% (216853/298536)
Region Coverage 70.87% (127721/180212)
Branch Coverage 64.61% (66150/102388)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 6, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 6, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 3de6d48 into apache:master Jun 6, 2025
26 of 29 checks passed
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Jun 27, 2025
@suxiaogang223
[improve](hive) Refactor csv reader (apache#50379)
02e8e1c 
- [x] Impl TextReader for reading hive text table.
- [x] Cleanup CsvReader
- [x] Add cases to test csv and text format

behavior changes:
1. Don't parse "\\N" or other string as null for hive OpenCsv table
In Hive's OpenCsv table, there is no definition for the representation
of null values. This behavior is different from the behavior of
importing CSV tables internally in Doris.
2. Fix the bug when reading complex type for hive OpenCsv table
The current code incorrectly uses the hive text format to parse the
complex types of Hive OpenCsv tables. This PR fixes this behavior and
uses the json format for parsing.
@suxiaogang223 suxiaogang223 mentioned this pull request Jun 27, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 left review comments

Copilot code review Copilot Copilot left review comments

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.1.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@suxiaogang223 @hello-stephen @doris-robot @morningman @CalvinKirs @starocean999 @hubgeter